Deep learning-based real-time detection and correction of compromised sensors in autonomous machines

ABSTRACT

A mechanism is described for facilitating deep learning-based real-time detection and correction of compromised sensors in autonomous machines according to one embodiment. An apparatus of embodiments, as described herein, includes detection and capturing logic to facilitate one or more sensors to capture one or more images of a scene, where an image of the one or more images is determined to be unclear, where the one or more sensors include one or more cameras. The apparatus further comprises classification and prediction logic to facilitate a deep learning model to identify, in real-time, a sensor associated with the image.

FIELD

Embodiments described herein relate generally to data processing andmore particularly to facilitate deep learning-based real-time detectionand correction of compromised sensors in autonomous machines.

BACKGROUND

Autonomous machines are expected to grow exponentially in the comingyears which, in turn, is likely to require sensors, such as cameras, tolead the growth in terms of facilitating various tasks, such asautonomous driving.

Conventional techniques use multiple sensors to attempt to applydata/sensor fusion for providing some redundancy to guarantee theaccuracy; however, these conventional techniques are severely limited inthat they are incapable of dealing with or getting around those sensorsthat provide low quality or misleading data.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by way oflimitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements.

FIG. 1 illustrates a computing device employing a sensor auto-checkingmechanism according to one embodiment.

FIG. 2 illustrates the sensor auto-checking mechanism of FIG. 1according to one embodiment.

FIG. 3A illustrates static inputs from multiple sensors according to oneembodiment.

FIG. 3B illustrates dynamic inputs from a single sensor according to oneembodiment.

FIG. 3C illustrates dynamic inputs from a single sensor according to oneembodiment.

FIG. 4A illustrates an architectural setup offering a transactionsequence for real-time detection and correction of compromised sensorsusing deep learning according to one embodiment.

FIG. 4B illustrates a method for real-time detection and correction ofcompromised sensors using deep learning according to one embodiment.

FIG. 5 illustrates a computer device capable of supporting andimplementing one or more embodiments according to one embodiment.

FIG. 6 illustrates an embodiment of a computing environment capable ofsupporting and implementing one or more embodiments according to oneembodiment.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth.However, embodiments, as described herein, may be practiced withoutthese specific details. In other instances, well-known circuits,structures and techniques have not been shown in detail in order not toobscure the understanding of this description.

Embodiments provide for a novel technique for deep learning-baseddetection, notification, correction of compromised sensors in autonomousmachines. In one embodiment, auto-checking may include one or more ofdetection of compromised sensors, issuing alerts to warn of thecompromised sensors, offering to fix, in real-time, any distortions ofcompromised sensors, and/or the like.

It is contemplated that embodiments are not limited to any number ortype of sensors; however, for the sake of brevity, clarity, and ease ofunderstanding, one or more cameras may be used as exemplary sensorsthroughout this document, but embodiments are not limited as such.

It is contemplated that terms like “request”, “query”, “job”, “work”,“work item”, and “workload” may be referenced interchangeably throughoutthis document. Similarly, an “application” or “agent” may refer to orinclude a computer program, a software application, a game, aworkstation application, etc., offered through an applicationprogramming interface (API), such as a free rendering API, such as OpenGraphics Library (OpenGL®), DirectX® 11, DirectX® 12, etc., where“dispatch” may be interchangeably referred to as “work unit” or “draw”and similarly, “application” may be interchangeably referred to as“workflow” or simply “agent”. For example, a workload, such as that of athree-dimensional (3D) game, may include and issue any number and typeof “frames” where each frame may represent an image (e.g., sailboat,human face). Further, each frame may include and offer any number andtype of work units, where each work unit may represent a part (e.g.,mast of sailboat, forehead of human face) of the image (e.g., sailboat,human face) represented by its corresponding frame. However, for thesake of consistency, each item may be referenced by a single term (e.g.,“dispatch”, “agent”, etc.) throughout this document.

In some embodiments, terms like “display screen” and “display surface”may be used interchangeably referring to the visible portion of adisplay device while the rest of the display device may be embedded intoa computing device, such as a smartphone, a wearable device, etc. It iscontemplated and to be noted that embodiments are not limited to anyparticular computing device, software application, hardware component,display device, display screen or surface, protocol, standard, etc. Forexample, embodiments may be applied to and used with any number and typeof real-time applications on any number and type of computers, such asdesktops, laptops, tablet computers, smartphones, head-mounted displaysand other wearable devices, and/or the like. Further, for example,rendering scenarios for efficient performance using this novel techniquemay range from simple scenarios, such as desktop compositing, to complexscenarios, such as 3D games, augmented reality applications, etc.

It is to be noted that terms or acronyms like convolutional neuralnetwork (CNN), CNN, neural network (NN), NN, deep neural network (DNN),DNN, recurrent neural network (RNN), RNN, and/or the like, may beinterchangeably referenced throughout this document. Further, terms like“autonomous machine” or simply “machine”, “autonomous vehicle” or simply“vehicle”, “autonomous agent” or simply “agent”, “autonomous device” or“computing device”, “robot”, and/or the like, may be interchangeablyreferenced throughout this document.

FIG. 1 illustrates a computing device 100 employing a sensorauto-checking mechanism (“auto-checking mechanism”) 110 according to oneembodiment. Computing device 100 represents a communication and dataprocessing device including or representing any number and type of smartdevices, such as (without limitation) smart command devices orintelligent personal assistants, home/office automation system, homeappliances (e.g., washing machines, television sets, etc.), mobiledevices (e.g., smartphones, tablet computers, etc.), gaming devices,handheld devices, wearable devices (e.g., smartwatches, smart bracelets,etc.), virtual reality (VR) devices, head-mounted display (HMDs),Internet of Things (IoT) devices, laptop computers, desktop computers,server computers, set-top boxes (e.g., Internet-based cable televisionset-top boxes, etc.), global positioning system (GPS)-based devices,etc.

In some embodiments, computing device 100 may include (withoutlimitation) autonomous machines or artificially intelligent agents, suchas a mechanical agents or machines, electronics agents or machines,virtual agents or machines, electro-mechanical agents or machines, etc.Examples of autonomous machines or artificially intelligent agents mayinclude (without limitation) robots, autonomous vehicles (e.g.,self-driving cars, self-flying planes, self-sailing boats, etc.),autonomous equipment (self-operating construction vehicles,self-operating medical equipment, etc.), and/or the like. Further,“autonomous vehicles” are not limed to automobiles but that they mayinclude any number and type of autonomous machines, such as robots,autonomous equipment, household autonomous devices, and/or the like, andany one or more tasks or operations relating to such autonomous machinesmay be interchangeably referenced with autonomous driving.

Further, for example, computing device 100 may include a computerplatform hosting an integrated circuit (“IC”), such as a system on achip (“SoC” or “SOC”), integrating various hardware and/or softwarecomponents of computing device 100 on a single chip.

As illustrated, in one embodiment, computing device 100 may include anynumber and type of hardware and/or software components, such as (withoutlimitation) graphics processing unit (“GPU” or simply “graphicsprocessor”) 114, graphics driver (also referred to as “GPU driver”,“graphics driver logic”, “driver logic”, user-mode driver (UMD), UMD,user-mode driver framework (UMDF), UMDF, or simply “driver”) 116,central processing unit (“CPU” or simply “application processor”) 112,memory 104, network devices, drivers, or the like, as well asinput/output (I/O) sources 108, such as touchscreens, touch panels,touch pads, virtual or regular keyboards, virtual or regular mice,ports, connectors, etc. Computing device 100 may include operatingsystem (OS) 106 serving as an interface between hardware and/or physicalresources of computing device 100 and a user.

It is to be appreciated that a lesser or more equipped system than theexample described above may be preferred for certain implementations.Therefore, the configuration of computing device 100 may vary fromimplementation to implementation depending upon numerous factors, suchas price constraints, performance requirements, technologicalimprovements, or other circumstances.

Embodiments may be implemented as any or a combination of: one or moremicrochips or integrated circuits interconnected using a parentboard,hardwired logic, software stored by a memory device and executed by amicroprocessor, firmware, an application specific integrated circuit(ASIC), and/or a field programmable gate array (FPGA). The terms“logic”, “module”, “component”, “engine”, and “mechanism” may include,by way of example, software or hardware and/or a combination thereof,such as firmware.

In one embodiment, as illustrated, auto-checking mechanism 110 may behosted by operating system 106 in communication with I/O source(s) 108of computing device 100. In another embodiment, auto-checking mechanism110 may be hosted or facilitated by graphics driver 116. In yet anotherembodiment, auto-checking mechanism 110 may be hosted by or part ofgraphics processing unit (“GPU” or simply graphics processor”) 114 orfirmware of graphics processor 114. For example, auto-checking mechanism110 may be embedded in or implemented as part of the processing hardwareof graphics processor 114. Similarly, in yet another embodiment,auto-checking mechanism 110 may be hosted by or part of centralprocessing unit (“CPU” or simply “application processor”) 112. Forexample, auto-checking mechanism 110 may be embedded in or implementedas part of the processing hardware of application processor 112.

In yet another embodiment, auto-checking mechanism 110 may be hosted byor part of any number and type of components of computing device 100,such as a portion of auto-checking mechanism 110 may be hosted by orpart of operating system 116, another portion may be hosted by or partof graphics processor 114, another portion may be hosted by or part ofapplication processor 112, while one or more portions of auto-checkingmechanism 110 may be hosted by or part of operating system 116 and/orany number and type of devices of computing device 100. It iscontemplated that embodiments are not limited to any particularimplementation or hosting of auto-checking mechanism 110 and that one ormore portions or components of auto-checking mechanism 110 may beemployed or implemented as hardware, software, or any combinationthereof, such as firmware.

Computing device 100 may host network interface(s) to provide access toa network, such as a LAN, a wide area network (WAN), a metropolitan areanetwork (MAN), a personal area network (PAN), Bluetooth, a cloudnetwork, a mobile network (e.g., 3rd Generation (3G), 4th Generation(4G), etc.), an intranet, the Internet, etc. Network interface(s) mayinclude, for example, a wireless network interface having antenna, whichmay represent one or more antenna(e). Network interface(s) may alsoinclude, for example, a wired network interface to communicate withremote devices via network cable, which may be, for example, an Ethernetcable, a coaxial cable, a fiber optic cable, a serial cable, or aparallel cable.

Embodiments may be provided, for example, as a computer program productwhich may include one or more machine-readable media having storedthereon machine-executable instructions that, when executed by one ormore machines such as a computer, network of computers, or otherelectronic devices, may result in the one or more machines carrying outoperations in accordance with embodiments described herein. Amachine-readable medium may include, but is not limited to, floppydiskettes, optical disks, CD-ROMs (Compact Disc-Read Only Memories), andmagneto-optical disks, ROMs, RAMs, EPROMs (Erasable Programmable ReadOnly Memories), EEPROMs (Electrically Erasable Programmable Read OnlyMemories), magnetic or optical cards, flash memory, or other type ofmedia/machine-readable medium suitable for storing machine-executableinstructions.

Moreover, embodiments may be downloaded as a computer program product,wherein the program may be transferred from a remote computer (e.g., aserver) to a requesting computer (e.g., a client) by way of one or moredata signals embodied in and/or modulated by a carrier wave or otherpropagation medium via a communication link (e.g., a modem and/ornetwork connection).

Throughout the document, term “user” may be interchangeably referred toas “viewer”, “observer”, “speaker”, “person”, “individual”, “end-user”,and/or the like. It is to be noted that throughout this document, termslike “graphics domain” may be referenced interchangeably with “graphicsprocessing unit”, “graphics processor”, or simply “GPU” and similarly,“CPU domain” or “host domain” may be referenced interchangeably with“computer processing unit”, “application processor”, or simply “CPU”.

It is to be noted that terms like “node”, “computing node”, “server”,“server device”, “cloud computer”, “cloud server”, “cloud servercomputer”, “machine”, “host machine”, “device”, “computing device”,“computer”, “computing system”, and the like, may be usedinterchangeably throughout this document. It is to be further noted thatterms like “application”, “software application”, “program”, “softwareprogram”, “package”, “software package”, and the like, may be usedinterchangeably throughout this document. Also, terms like “job”,“input”, “request”, “message”, and the like, may be used interchangeablythroughout this document.

FIG. 2 illustrates sensor auto-checking mechanism 110 of FIG. 1according to one embodiment. For brevity, many of the details alreadydiscussed with reference to FIG. 1 are not repeated or discussedhereafter. In one embodiment, auto-checking mechanism 110 may includeany number and type of components, such as (without limitations):detection and capturing logic 201; concatenation logic 203; training andinference logic 205; communication/compatibility logic 209; andclassification and prediction logic 207.

Computing device 100 (also interchangeably referenced as “autonomousmachine” throughout the document) is further shown to include userinterface 219 (e.g., graphical user interface (GUI)-based userinterface, Web browser, cloud-based platform user interface, softwareapplication-based user interface, other user or application programminginterfaces (APIs), etc.). Computing device 100 may further include I/Osource(s) 108 having capturing/sensing component(s) 231, such ascamera(s) A 242A, B 242B, C 242C, D 242D (e.g., Intel® RealSense™camera), sensors, microphone(s) 241, etc., and output component(s) 233,such as display device(s) or simply display(s) 244 (e.g., integraldisplays, tensor displays, projection screens, display screens, etc.),speaker devices(s) or simply speaker(s) 243, etc.

Computing device 100 is further illustrated as having access to and/orbeing in communication with one or more database(s) 225 and/or one ormore of other computing devices over one or more communication medium(s)230 (e.g., networks such as a cloud network, a proximity network, theInternet, etc.).

In some embodiments, database(s) 225 may include one or more of storagemediums or devices, repositories, data sources, etc., having any amountand type of information, such as data, metadata, etc., relating to anynumber and type of applications, such as data and/or metadata relatingto one or more users, physical locations or areas, applicable laws,policies and/or regulations, user preferences and/or profiles, securityand/or authentication data, historical and/or preferred details, and/orthe like.

As aforementioned, computing device 100 may host I/O sources 108including capturing/sensing component(s) 231 and output component(s)233. In one embodiment, capturing/sensing component(s) 231 may include asensor array including, but not limited to, microphone(s) 241 (e.g.,ultrasound microphones), camera(s) 242A-242D (e.g., two-dimensional (2D)cameras, three-dimensional (3D) cameras, infrared (IR) cameras,depth-sensing cameras, etc.), capacitors, radio components, radarcomponents, scanners, and/or accelerometers, etc. Similarly, outputcomponent(s) 233 may include any number and type of speaker(s) 243,display device(s) 244 (e.g., screens, projectors, light-emitting diodes(LEDs)), and/or vibration motors, etc.

For example, as illustrated, capturing/sensing component(s) 231 mayinclude any number and type of microphones(s) 241, such as multiplemicrophones or a microphone array, such as ultrasound microphones,dynamic microphones, fiber optic microphones, laser microphones, etc. Itis contemplated that one or more of microphone(s) 241 serve as one ormore input devices for accepting or receiving audio inputs (such ashuman voice) into computing device 100 and converting this audio orsound into electrical signals. Similarly, it is contemplated that one ormore of camera(s) 242A-242D serve as one or more input devices fordetecting and capturing of image and/or videos of scenes, objects, etc.,and provide the captured data as video inputs into computing device 100.

It is contemplated that embodiments are not limited to any number ortype of microphone(s) 241, camera(s) 242A-242D, speaker(s) 243,display(s) 244, etc. For example, as facilitated by detection andcapturing logic 201, one or more of microphone(s) 241 may be used todetect speech or sound simultaneously from multiple users or speakers,such as speaker 250. Similarly, as facilitated by detection andcapturing logic 201, one or more of camera(s) 242A-242D may be used tocapture images or videos of a geographic location (such as a room) andits contents (e.g., furniture, electronic devices, humans, animals,plats, etc.) and form a set of images or a video stream form thecaptured data for further processing by auto-checking mechanism 110 atcomputing device 100.

Similarly, as illustrated, output component(s) 233 may include anynumber and type of speaker(s) 243 to serve as output devices foroutputting or giving out audio from computing device 100 for any numberor type of reasons, such as human hearing or consumption. For example,speaker(s) 243 work the opposite of microphone(s) 241 where speaker(s)243 convert electric signals into sound.

As mentioned previously, embodiments are not limited to any number ortype of sensors that are part of, embedded in, or coupled tocapturing/sensing component(s) 231, such as microphones 241, cameras242A-242D, and/or the like. In other words, embodiments are applicableto and compatible with any number type of sensors; however, cameras242A-242D are used as examples throughout this document for the purposesof discussion with brevity and clarity. Similarly, embodiments areapplicable with all types and manner of cameras and thus cameras242A-242D do not have to be of a certain type.

As aforementioned, with the growth of autonomous machines, such asself-driving vehicles, drones, household appliances, etc., sensors ofall sorts are expected to lead the way to influence and facilitatecertain tasks that are essential for the viability of autonomousmachines, such as sensors serving as the eyes behind the wheel in caseof self-driving vehicles. As such, data quality becomes a criticalfactor when dealing with autonomous machines for any number of reasons,such as safety, security, trust, etc.; particularly, in life-and-deathsituations, business environments, etc.

It is contemplated that high-quality data can ensure the artificialintelligence (AI) of an autonomous machine, such as computing device100, receives high-quality inputs (e.g., images, videos, etc.) foroutputting high-quality performance. It is further contemplated thateven if one of the sensors, such as cameras 242A-242D, is defective ornot performing to its full potential (such as due to mud on its lens ortoo much fog, etc.), the overall performance of computing device 100could suffer as its accuracy is compromised.

For example, auto-checking mechanism 110 provides for a novel techniquefor filtering through cameras 242A-242D to detect any abnormalities withany one or more of cameras 242A-242D that may be responsible or have thepotential for offering less than high-quality inputs, where suchabnormalities in or with cameras 242A-242D may include (withoutlimitation) dirt/mud on lenses, obstacles before lenses, occultations(e.g., fog), physical damage technical issues, and/or the like.

Conventional techniques are incapable of detecting such abnormalitiesand thus cannot guarantee accuracy of data being collected by sensors ofautonomous machines, which often leads to low-quality data or evenmisleading data.

Embodiments provide for a novel technique for real-time detection ofabnormalities with sensors, such as cameras 242A-242D, issuance of alertor warning, as necessitated, and fixing or repairing of suchabnormalities. In one embodiment, auto-checking mechanism 110 providesfor a novel technique for sensors automatic checking (SAC) for detectionand checking on the status of each sensor, such as cameras 242A-242D, ina system, such as autonomous machine 100, to ensure all sensors areworking well before any tasks are undertaken (such as prior to driving aself-driving car) and continue to check on the sensors to make certainthey go on working or in case of any abnormalities, they are fixed inreal-time during performance of any of the tasks (such as driving).

In one embodiment, auto-checking mechanism 110 uses deep learning ofautonomous machine 100 to ensure, real-time, cameras 242A-242D and anyother sensors are in working condition or that they are at leastimmediately attended to and fixed in case of any issues. As will befurther described later in this document, auto-checking mechanism 110may use deep neural networks (DNNs), such as convolutional deep learningclassifiers of convolutional neural networks (CNNs), to continuously andaccurately check on the real-time status of cameras 242A-242D and othersensors and then use the training data to detect and predict which ofcameras 242A-242D or other sensors may possibly be broken or out ofcommission.

One of the major weaknesses with conventional techniques is when acamera lens gets covered with debris, such as dirt, stain, mud, etc.,because when that happens, no matter the level or about of debris, thereremains no ability for the camera to detect or capture.

Embodiments provide for the use of deep learning on autonomous machines,such as autonomous machine 100, to handle complex matters, such as incase of stain or mud on the lens of a camera, such as camera 242A, thisobstruction may be continuously observed including considering anymovements or changes associated with camera 242A, the stain, and/or thescene. This is detected and observed in real-time so that the defectiveor obstructed camera 242A may fixed.

In one embodiment, detection and capturing logic 201 of auto-checkingmechanism 110 may be used to trigger one or more cameras 242A-242D,located at various positions, to capture one or more scenes in front ofthem. It is contemplated that in some embodiments, as illustrated withrespect to FIG. 3A, there may be multiple cameras 242A-242D fixed intheir locations capturing static inputs, such as capturing the scenefrom different angles at the same time. Similarly, as illustrated inFIG. 3B, in another embodiment, a single camera, such as camera 242A,may be used to capture dynamic inputs, such as capturing the scene fromthe same angel at different points in time. In yet another input, asillustrated with respect to FIG. 3C, the stain or debris itself may bedynamic or moving and so a camera, such as camera 242B, may be used tocapture the scene while capturing the movement of the debris.

As discussed above, sensors of capturing/sensing components 231 are notmerely limited to any number or type of cameras 242A-242D or microphones241 and that sensors may further include other sensors, such as LightDetection and Ranging (LiDAR) sensors, ultrasonic sensors, and anynumber and type of other sensors mentioned or described throughout thisdocument and that any input from such sensors may be inputted into aneural network, such as to a softmax layer of a CNN, for classificationpurposes.

Referring back to auto-checking mechanism 110, as one or more cameras242A-242D are capturing a scene, any internal or external issues withany of cameras 242A-242D may also be detected, where internal issuesinclude any physical defect (such as part of the lens or camera isbroken) or technical issues (such as camera stops working), whileexternal issues relate to any form of obstruction, such as snow, trees,dirt, mud, debris, persons, animals, etc., that could be on the lens orin view of the lens blocking the view of the scene.

For example, if some mud is found on the lens of camera 242A, detectionand capturing logic 201 may be triggered to detect that mud or at leastthat the view from camera 242A is somehow blocked. In case of anymovements associated with any of camera 242A, the scene (such as peoplemoving, ocean waves, traffic movement, etc.), and/or the mud itself(such as flowing downwards or in the direct of the wind, etc.),detection and capturing logic 201 may collect such data that includesinformation relating to the blockage of the view from camera 242A aswell as any one or more movements mentioned above.

Once the data is collected by detection and capturing logic 201, it isthen forwarded on to concatenation logic 203 as inputs. As mentionedabove, embodiments are not limited to camera inputs and that such inputsmay come from other sensors and include LiDAR inputs, radar inputs,microphone inputs, and/or the like, where there may be some degree ofoverlapping in detections of the same object from such sensors. In oneembodiment, in case of multiple inputs from multiple sensors, such astwo or more of cameras 242A-242D at the same time or over differentpoints in time and/or from the same or different angles, and/or the samesensor, such as camera 242A, over multiple points in time and/or fromthe same or different angles, concatenation logic 203 may then betriggered to concatenate (or concat) these inputs into a single input.

In one embodiment, any concatenated input of multiple inputs may then beforwarded on to a deep learning neural network model, such as a CNN, fortraining and interference by training and inference logic 205. In oneembodiment, concatenation logic 203 performs concatenation outside of orprior to the data being handled by the deep learning model so that thereis better and flexibility to set their orders to further benefit thetraining process. It is contemplated that embodiments are not limited toany number and type of deep learning models such that a CNN may be anysort or type of CNN commonly used, such as AlexNet, GoogLeNet, RESNET,and/or the like.

It is contemplated that a deep learning neural network/model, such as aCNN, refers to a combination of artificial neural network for analyzing,training, and inferring any range of input data. For example, a CNN ismuch faster and may necessitate relatively less processing of datacompared to conventional algorithms. It is further contemplated thatonce the input data is received at a CNN, the data may then be processedthrough layers, such as convolutional layer, a pooling layer, aRectified Linear Unit (ReLU) layer, a fully connected layer, aloss/output layer, etc., where each layer performs specific processingtasks for training and inferring purposes.

For example, a convolutional layer may be regarded as a core layerhaving a number of learnable filters or kernels with receptive fields,extending through the full depth of the input volume. This convolutionallayer is where the processing of any data from the inputs may getstarted and move on to another layer, such as pooling layer, where aform of non-linear down-sampling is performed, where, for example, thesenon-linear down-sampling functions may implement pooling, such as maxpooling. Similarly, the data is further process and trained at ReLUlayer, which applies the non-saturating activation function to increasenonlinear properties of the decision function and the network withoutimpacting the receptive fields of the convolution layer.

Although embodiments are not limited to any number or types of layers ofa neural network, such as a CNN, the training process may continue withthe fully connected layer where after several convolutional and poolinglayers, high-level reasoning is provided. In other words, a CNN mayreceive input data and perform feature mapping, sampling, convolutions,sub-sampling, followed by output results.

For example, a loss/output layer may specify how training penalizes thedeviation between the predicted labels and true labels, where thisloss/output layer may be regarded as the last layer in the CNN. Forexample, softmax loss may be used for predicting a single class ofmutually exclusive classes. Further, in one embodiment, classificationand prediction logic 207 for example, softmax and classification layersof loss/output layer may be used for classification and predictionpurposed where the two layers are generated by softmax layer andclassification layer functions, respectively.

In one embodiment, after having process all the data from inputs fortraining and inference, classification and prediction logic 207 may thenbe used to identify which of the sensors, such as cameras 242A-242D, mayhave problems. Once identified, classification and prediction logic 207may put out a notification regarding the bad one of cameras 242A-242D,such as display the notification at display device(s) 244, sound itthrough speaker device(s) 243, etc. In one embodiment, this notificationmay then be used, such as by a user, to get to the defective one ofcameras 242A-242D and fix the problem, such as wipe off the mud from thelens, manually or automatically fix any technical glitch with the lens,replace the defective one of cameras 242-242D with another one, and/orthe like.

In one embodiment, certain labels may be used for notification purposes,such as label: 0 may mean all sensors are fine, while label: 1 may meanfirst sensor is damaged, label: 2 may indicate second sensor is damaged,label: 3 may mean third sensor is damaged, label: 4 may indicate fourthsensor is damaged, and so on. Similarly, label: 1 may indicate firstsensor is fine, label: 2 may indicate second sensor is fine, and/or thelike. It is contemplated that embodiments are not limited to any form ofnotification and that anyone or combination of words, numbers, images,videos, audio, etc., may be used to convey the results of sensors beingworking well or not.

Further, for example, with a single input data layer associated witheach of cameras 242A-242D, a number of channels, such as 12 (3*4=12)channels in case of four cameras 242A-242D, may provide for all the dataof four images corresponding to four cameras 242A-242D, where this datamay be loaded randomly by disrupting the order of channels. Using thisdata, a deep learning model, such as a CNN, may calculate loss (duringtraining) and accuracy (during validation), so when comes prediction,there may not be a need to use labels. In some embodiments, trainingdata may include a large sample of images, such as thousands or tens ofthousands of sample images per camera 242A-242D, while validation datamay also include a large sample of images, such as hundreds or thousandsof samples images per camera 242A-242D, and/or the like, to provide fora robust training/inferencing of data as facilitated by training andinference data, which is then followed by accurate results, includingidentifications, predictions, etc., as facilitated by classification andprediction logic 207.

Capturing/sensing component(s) 231 may further include any number andtype of camera(s) 242A, 242B, 242C, 242D, such as depth-sensing camerasor capturing devices (e.g., Intel® RealSense™ depth-sensing camera) thatare known for capturing still and/or video red-green-blue (RGB) and/orRGB-depth (RGB-D) images for media, such as personal media. Such images,having depth information, have been effectively used for variouscomputer vision and computational photography effects, such as (withoutlimitations) scene understanding, refocusing, composition,cinema-graphs, etc. Similarly, for example, displays may include anynumber and type of displays, such as integral displays, tensor displays,stereoscopic displays, etc., including (but not limited to) embedded orconnected display screens, display devices, projectors, etc.

Capturing/sensing component(s) 231 may further include one or more ofvibration components, tactile components, conductance elements,biometric sensors, chemical detectors, signal detectors,electroencephalography, functional near-infrared spectroscopy, wavedetectors, force sensors (e.g., accelerometers), illuminators,eye-tracking or gaze-tracking system, head-tracking system, etc., thatmay be used for capturing any amount and type of visual data, such asimages (e.g., photos, videos, movies, audio/video streams, etc.), andnon-visual data, such as audio streams or signals (e.g., sound, noise,vibration, ultrasound, etc.), radio waves (e.g., wireless signals, suchas wireless signals having data, metadata, signs, etc.), chemicalchanges or properties (e.g., humidity, body temperature, etc.),biometric readings (e.g., figure prints, etc.), brainwaves, braincirculation, environmental/weather conditions, maps, etc. It iscontemplated that “sensor” and “detector” may be referencedinterchangeably throughout this document. It is further contemplatedthat one or more capturing/sensing component(s) 231 may further includeone or more of supporting or supplemental devices for capturing and/orsensing of data, such as illuminators (e.g., IR illuminator), lightfixtures, generators, sound blockers, etc.

It is further contemplated that in one embodiment, capturing/sensingcomponent(s) 231 may further include any number and type of contextsensors (e.g., linear accelerometer) for sensing or detecting any numberand type of contexts (e.g., estimating horizon, linear acceleration,etc., relating to a mobile computing device, etc.). For example,capturing/sensing component(s) 231 may include any number and type ofsensors, such as (without limitations): accelerometers (e.g., linearaccelerometer to measure linear acceleration, etc.); inertial devices(e.g., inertial accelerometers, inertial gyroscopes,micro-electro-mechanical systems (MEMS) gyroscopes, inertial navigators,etc.); and gravity gradiometers to study and measure variations ingravitation acceleration due to gravity, etc.

Further, for example, capturing/sensing component(s) 231 may include(without limitations): audio/visual devices (e.g., cameras, microphones,speakers, etc.); context-aware sensors (e.g., temperature sensors,facial expression and feature measurement sensors working with one ormore cameras of audio/visual devices, environment sensors (such as tosense background colors, lights, etc.); biometric sensors (such as todetect fingerprints, etc.), calendar maintenance and reading device),etc.; global positioning system (GPS) sensors; resource requestor;and/or TEE logic. TEE logic may be employed separately or be part ofresource requestor and/or an I/O subsystem, etc. Capturing/sensingcomponent(s) 231 may further include voice recognition devices, photorecognition devices, facial and other body recognition components,voice-to-text conversion components, etc.

Similarly, output component(s) 233 may include dynamic tactile touchscreens having tactile effectors as an example of presentingvisualization of touch, where an embodiment of such may be ultrasonicgenerators that can send signals in space which, when reaching, forexample, human fingers can cause tactile sensation or like feeling onthe fingers. Further, for example and in one embodiment, outputcomponent(s) 233 may include (without limitation) one or more of lightsources, display devices and/or screens, audio speakers, tactilecomponents, conductance elements, bone conducting speakers, olfactory orsmell visual and/or non/visual presentation devices, haptic or touchvisual and/or non-visual presentation devices, animation displaydevices, biometric display devices, X-ray display devices,high-resolution displays, high-dynamic range displays, multi-viewdisplays, and head-mounted displays (HMDs) for at least one of virtualreality (VR) and augmented reality (AR), etc.

It is contemplated that embodiment are not limited to any particularnumber or type of use-case scenarios, architectural placements, orcomponent setups; however, for the sake of brevity and clarity,illustrations and descriptions are offered and discussed throughout thisdocument for exemplary purposes but that embodiments are not limited assuch. Further, throughout this document, “user” may refer to someonehaving access to one or more computing devices, such as computing device100, and may be referenced interchangeably with “person”, “individual”,“human”, “him”, “her”, “child”, “adult”, “viewer”, “player”, “gamer”,“developer”, programmer”, and/or the like.

Communication/compatibility logic 209 may be used to facilitate dynamiccommunication and compatibility between various components, networks,computing devices, database(s) 225, and/or communication medium(s) 230,etc., and any number and type of other computing devices (such aswearable computing devices, mobile computing devices, desktop computers,server computing devices, etc.), processing devices (e.g., centralprocessing unit (CPU), graphics processing unit (GPU), etc.),capturing/sensing components (e.g., non-visual data sensors/detectors,such as audio sensors, olfactory sensors, haptic sensors, signalsensors, vibration sensors, chemicals detectors, radio wave detectors,force sensors, weather/temperature sensors, body/biometric sensors,scanners, etc., and visual data sensors/detectors, such as cameras,etc.), user/context-awareness components and/oridentification/verification sensors/devices (such as biometricsensors/detectors, scanners, etc.), memory or storage devices, datasources, and/or database(s) (such as data storage devices, hard drives,solid-state drives, hard disks, memory cards or devices, memorycircuits, etc.), network(s) (e.g., Cloud network, Internet, Internet ofThings, intranet, cellular network, proximity networks, such asBluetooth, Bluetooth low energy (BLE), Bluetooth Smart, Wi-Fi proximity,Radio Frequency Identification, Near Field Communication, Body AreaNetwork, etc.), wireless or wired communications and relevant protocols(e.g., Wi-Fi®, WiMAX, Ethernet, etc.), connectivity and locationmanagement techniques, software applications/websites, (e.g., socialand/or business networking websites, business applications, games andother entertainment applications, etc.), programming languages, etc.,while ensuring compatibility with changing technologies, parameters,protocols, standards, etc.

Throughout this document, terms like “logic”, “component”, “module”,“framework”, “engine”, “tool”, “circuitry”, and/or the like, may bereferenced interchangeably and include, by way of example, software,hardware, and/or any combination of software and hardware, such asfirmware. In one example, “logic” may refer to or include a softwarecomponent that is capable of working with one or more of an operatingsystem, a graphics driver, etc., of a computing device, such ascomputing device 100. In another example, “logic” may refer to orinclude a hardware component that is capable of being physicallyinstalled along with or as part of one or more system hardware elements,such as an application processor, a graphics processor, etc., of acomputing device, such as computing device 100. In yet anotherembodiment, “logic” may refer to or include a firmware component that iscapable of being part of system firmware, such as firmware of anapplication processor or a graphics processor, etc., of a computingdevice, such as computing device 100.

Further, any use of a particular brand, word, term, phrase, name, and/oracronym, such as “sensors”, “cameras”, “autonomous machines”, “sensorautomatic checking”, “deep learning”, “convolution neural network”,“concatenating”, “training”, “inferencing”, “classifying”, “predicting”,“RealSense™ camera”, “real-time”, “automatic”, “dynamic”, “userinterface”, “camera”, “sensor”, “microphone”, “display screen”,“speaker”, “verification”, “authentication”, “privacy”, “user”, “userprofile”, “user preference”, “sender”, “receiver”, “personal device”,“smart device”, “mobile computer”, “wearable device”, “IoT device”,“proximity network”, “cloud network”, “server computer”, etc., shouldnot be read to limit embodiments to software or devices that carry thatlabel in products or in literature external to this document.

It is contemplated that any number and type of components may be addedto and/or removed from auto-checking mechanism 110 to facilitate variousembodiments including adding, removing, and/or enhancing certainfeatures. For brevity, clarity, and ease of understanding ofauto-checking mechanism 110, many of the standard and/or knowncomponents, such as those of a computing device, are not shown ordiscussed here. It is contemplated that embodiments, as describedherein, are not limited to any technology, topology, system,architecture, and/or standard and are dynamic enough to adopt and adaptto any future changes.

FIG. 3A illustrates static inputs from multiple sensors according to oneembodiment and as previously described with reference to FIG. 2. Forbrevity, many of the details previously discussed with reference toFIGS. 1-2 may not be discussed or repeated hereafter.

In the illustrated embodiment, four images A 301, B 303, C 305, and D307 of a scene are shown as captured by four cameras A 242A, B 242B,C242C, and D 242D, respectively, of FIG. 2, where these multiple images301-307 are based on static data captured by four cameras 242A-242D overa length of time. For example, sensors, such as cameras 242A-242D,radars, etc., may be used to capture similar data for the same purposeof sensing, such as for the automated driving vehicles to be aware ofthe objects near or around them.

In this embodiment, to simply for design and test, four cameras242A-242D are shown as capturing four images 301-307 of the same sceneand at the same time, but from different angles and/or positions.Further, as illustrated, one of the images, such as image 301, shows thecorresponding camera 242A having clarity issues, such as due to somesort of stain 309 (e.g., mud, dirt, debris, etc.) on the lens of camera242A. It is contemplated that such issues can lead to a great deal ofissues when dealing with autonomous machines, such as a self-drivingvehicle.

In one embodiment, as discussed with reference to FIG. 2, by collectinga large amount of data, such as thousands of data inputs, and using themas training data, validation data, etc., in deep learning models, suchas CNNs, as facilitated by auto-checking mechanism 110 of FIG. 1 allowsfor real-time detection of stain 309. This real-time detection thenallows for real-time notification as real-time correction of stain 309so that any defects with regard to camera 242A may be fixed and allcameras 242A-242D may function to their potential and collect data tomake the use of autonomous machines, such as autonomous machine 100 ofFIG. 1, safe, secure, and efficient.

FIG. 3B illustrates dynamic inputs from a single sensor according to oneembodiment and as previously described with reference to FIG. 2. Forbrevity, many of the details previously discussed with reference toFIGS. 1-3A may not be discussed or repeated hereafter.

In this illustrated embodiment, a single sensor, such as camera D 242Dof FIG. 2, may be used to capture four images A 311, B 313, C 315, D 317of a single scene, but with different timestamps, such as at differentpoints in time. In this illustrated pattern, capturing the scene atdifferent points in time show the scene as moving, such as from right toleft, while stain 319 is shown as being placed in one location, such asin one spot on the lens of camera D 242D.

As revealed in four images 311-317, as time goes by, object 321 (e.g.,book) as captured by camera 242D seems to be moving (such as from rightto left), while stain 319 is fixed (or in real sense, moving slowly onin a different pattern as illustrated in the embodiment of FIG. 3C). Asdescribed with reference to FIGS. 2 and 3A, several thousands of imagesare collected and inputted into a deep learning model for training andvalidation purposes, which then results in testing of the deep learningmodel. Once tested, the deep learning model may be used for real-timeidentification and correction of problems with sensors, such as stain319 on camera 242D.

FIG. 3C illustrates dynamic inputs from a single sensor according to oneembodiment and as previously described with reference to FIG. 2. Forbrevity, many of the details previously discussed with reference toFIGS. 1-3B may not be discussed or repeated hereafter.

In one embodiment, as described with reference to FIG. 3B in terms ofhaving dynamic inputs through a single sensor, in this illustratedembodiment, a single sensor, such as camera B 242B captures four imagesA 331, B 333, C 335, D 337 of a single scene, where stain 339 on thelens of camera 242B is shown as moving with object 341 (e.g., book) inthe background scene. For example, stain 339 be a piece of mud on thelens of camera 242B that over time draws downward due to gravity orsideways due to winds, movements of camera 242B, and/or the like.

In one embodiment, as previously described, this data relating to stain339 and its movements may be captured through one or more sensors, suchas camera 242B itself, and inputted into a trained deep learning neuralnetwork/model, such as a CNN, which then predicts and provides, inreal-time, the exact location of stain 339, the sensor impacted by stain339, such as camera 242B, and how to correct this issue, such as how toremove stain 339 from the lens of camera 242B.

In one embodiment, this training of deep learning neural networks/modelsis achieved through inputs of collection of (thousands) of such inputsas examples for training and validation of data and testing of deeplearning models. For example, a deep learning model may first extractfeatures of all sensors, such as camera 242B, using deep learning neuralnetworks, such as a CNN, and then fuse the data and use a classifier toidentify the sensors that are problematic, such as camera 242B. Forexample, camera 242B may be assigned a label, such as label 2: secondsensor is damaged, and/or the like.

FIG. 4A illustrates an architectural setup 400 offering a transactionsequence for real-time detection and correction of compromised sensorsusing deep learning according to one embodiment. For brevity, many ofthe details previously discussed with reference to FIGS. 1-3C may not bediscussed or repeated hereafter. Any processes or transactions may beperformed by processing logic that may comprise hardware (e.g.,circuitry, dedicated logic, programmable logic, etc.), software (such asinstructions run on a processing device), or a combination thereof, asfacilitated by auto-checking mechanism 110 of FIG. 1. Any processes ortransactions associated with this illustration may be illustrated orrecited in linear sequences for brevity and clarity in presentation;however, it is contemplated that any number of them can be performed inparallel, asynchronously, or in different orders.

In one embodiment, the transaction sequence at architectural setup 400begins at 401 with inputting of captured data from one or more sensorsof sequential multiple times for concatenation prior to inputting itinto a deep learning model, such as deep learning model 421. This datamay include multiple inputs based on data captured by any number andtype of sensors, such as cameras, LiDARs, radars, etc., while there issome degree of overlapping in their detections (such as when detectingthe same object in a scene). At 403, as described with reference toconcatenation logic 203 of FIG. 2, data from these inputs sent forconcatenation such that these multiple inputs are then concatenated intoa single input and sent to data learning model 421 for training 405 andinferencing 407.

It is contemplated that in one embodiment, concatenation is performedoutside of or prior to sending the data to deep learning model 421 sothat there is better flexibility with respect to setting of orders togain maximum benefit from training 405. As illustrated, in oneembodiment, inferencing 407 may be part of training 405 or, in anotherembodiment, inferencing 407 and training 405 may be performedseparately.

In one embodiment, once inputted into deep learning model 421, it isthen inputted into and processed by CNN 409, where the processing of thedata passes through multiple layers as further described with referenceto FIG. 2. For example, at classification layer 411 may include a commonclassification layer, such as fully connected layers, softmax layer,and/or the like, as further described with reference to FIG. 2.

In one embodiment, the transaction sequence as offered by architecturalsetup 400 may continue with results 413 obtained through an outputlayer, where results 413 may identify or predict whether one or moresensors are technically defective or obstructed by an object or debrisor not working for any reason. Once results 413 have been obtained,various labels 415 are compared to determine the loss and theappropriate label to offer to the user regarding the one or moredefective sensors. The transaction sequence may continue with backpropagation 417 of data and consequently, more weight updates 419 areperformed, all at deep learning model 421.

FIG. 4B illustrates a method 450 for real-time detection and correctionof compromised sensors using deep learning according to one embodiment.For brevity, many of the details previously discussed with reference toFIGS. 1-4A may not be discussed or repeated hereafter. Any processes ortransactions may be performed by processing logic that may comprisehardware (e.g., circuitry, dedicated logic, programmable logic, etc.),software (such as instructions run on a processing device), or acombination thereof, as facilitated by auto-checking mechanism 110 ofFIG. 1. Any processes or transactions associated with this illustrationmay be illustrated or recited in linear sequences for brevity andclarity in presentation; however, it is contemplated that any number ofthem can be performed in parallel, asynchronously, or in differentorders.

Method 450 begins at block 451 with detection of data including one ormore images of a scene captured by one or more sensors (e.g., cameras)at the same time or over a period of time, where in case of this databeing spread over multiple inputs, these multiple inputs are offered forconcatenation. At block 453, these multiple inputs are concatenated intoa single input of data and offered to a deep learning model for furtherprocessing, such as training, inferencing, validation, etc. At block455, this data is received at the deep learning model for training andinferencing, where the deep learning model includes a neural network(such as a CNN) having multiple processing layers.

It is contemplated and as discussed with reference to FIG. 2, the datapassing through training and inferencing stages may be processed andmodified at several levels, including at the CNN which may includemultiple processing layers of its own. In one embodiment, at block 457,a trained deep learning model classifies the data and predicts theresults based on all the processing and classification. For example, theprediction of results may indicate and identify, in real-time, whetherany of the one or more sensors is defective or obstructed so thatdefective or obstructed sensor may be fixed in real-time.

FIG. 5 illustrates a computing device 500 in accordance with oneimplementation. The illustrated computing device 500 may be same as orsimilar to computing device 100 of FIG. 1. The computing device 500houses a system board 502. The board 502 may include a number ofcomponents, including but not limited to a processor 504 and at leastone communication package 506. The communication package is coupled toone or more antennas 516. The processor 504 is physically andelectrically coupled to the board 502.

Depending on its applications, computing device 500 may include othercomponents that may or may not be physically and electrically coupled tothe board 502. These other components include, but are not limited to,volatile memory (e.g., DRAM) 508, non-volatile memory (e.g., ROM) 509,flash memory (not shown), a graphics processor 512, a digital signalprocessor (not shown), a crypto processor (not shown), a chipset 514, anantenna 516, a display 518 such as a touchscreen display, a touchscreencontroller 520, a battery 522, an audio codec (not shown), a video codec(not shown), a power amplifier 524, a global positioning system (GPS)device 526, a compass 528, an accelerometer (not shown), a gyroscope(not shown), a speaker 530, cameras 532, a microphone array 534, and amass storage device (such as hard disk drive) 510, compact disk (CD)(not shown), digital versatile disk (DVD) (not shown), and so forth).These components may be connected to the system board 502, mounted tothe system board, or combined with any of the other components.

The communication package 506 enables wireless and/or wiredcommunications for the transfer of data to and from the computing device500. The term “wireless” and its derivatives may be used to describecircuits, devices, systems, methods, techniques, communicationschannels, etc., that may communicate data through the use of modulatedelectromagnetic radiation through a non-solid medium. The term does notimply that the associated devices do not contain any wires, although insome embodiments they might not. The communication package 506 mayimplement any of a number of wireless or wired standards or protocols,including but not limited to Wi-Fi (IEEE 802.11 family), WiMAX (IEEE802.16 family), IEEE 802.20, long term evolution (LTE), Ev-DO, HSPA+,HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, Bluetooth, Ethernetderivatives thereof, as well as any other wireless and wired protocolsthat are designated as 3G, 4G, 5G, and beyond. The computing device 500may include a plurality of communication packages 506. For instance, afirst communication package 506 may be dedicated to shorter rangewireless communications such as Wi-Fi and Bluetooth and a secondcommunication package 506 may be dedicated to longer range wirelesscommunications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, andothers.

The cameras 532 including any depth sensors or proximity sensor arecoupled to an optional image processor 536 to perform conversions,analysis, noise reduction, comparisons, depth or distance analysis,image understanding and other processes as described herein. Theprocessor 504 is coupled to the image processor to drive the processwith interrupts, set parameters, and control operations of imageprocessor and the cameras. Image processing may instead be performed inthe processor 504, the graphics CPU 512, the cameras 532, or in anyother device.

In various implementations, the computing device 500 may be a laptop, anetbook, a notebook, an ultrabook, a smartphone, a tablet, a personaldigital assistant (PDA), an ultra mobile PC, a mobile phone, a desktopcomputer, a server, a set-top box, an entertainment control unit, adigital camera, a portable music player, or a digital video recorder.The computing device may be fixed, portable, or wearable. In furtherimplementations, the computing device 500 may be any other electronicdevice that processes data or records data for processing elsewhere.

Embodiments may be implemented using one or more memory chips,controllers, CPUs (Central Processing Unit), microchips or integratedcircuits interconnected using a motherboard, an application specificintegrated circuit (ASIC), and/or a field programmable gate array(FPGA). The term “logic” may include, by way of example, software orhardware and/or combinations of software and hardware.

References to “one embodiment”, “an embodiment”, “example embodiment”,“various embodiments”, etc., indicate that the embodiment(s) sodescribed may include particular features, structures, orcharacteristics, but not every embodiment necessarily includes theparticular features, structures, or characteristics. Further, someembodiments may have some, all, or none of the features described forother embodiments.

In the following description and claims, the term “coupled” along withits derivatives, may be used. “Coupled” is used to indicate that two ormore elements co-operate or interact with each other, but they may ormay not have intervening physical or electrical components between them.

As used in the claims, unless otherwise specified, the use of theordinal adjectives “first”, “second”, “third”, etc., to describe acommon element, merely indicate that different instances of likeelements are being referred to, and are not intended to imply that theelements so described must be in a given sequence, either temporally,spatially, in ranking, or in any other manner.

The drawings and the forgoing description give examples of embodiments.Those skilled in the art will appreciate that one or more of thedescribed elements may well be combined into a single functionalelement. Alternatively, certain elements may be split into multiplefunctional elements. Elements from one embodiment may be added toanother embodiment. For example, orders of processes described hereinmay be changed and are not limited to the manner described herein.Moreover, the actions of any flow diagram need not be implemented in theorder shown; nor do all of the acts necessarily need to be performed.Also, those acts that are not dependent on other acts may be performedin parallel with the other acts. The scope of embodiments is by no meanslimited by these specific examples. Numerous variations, whetherexplicitly given in the specification or not, such as differences instructure, dimension, and use of material, are possible. The scope ofembodiments is at least as broad as given by the following claims.

Embodiments may be provided, for example, as a computer program productwhich may include one or more transitory or non-transitorymachine-readable storage media having stored thereon machine-executableinstructions that, when executed by one or more machines such as acomputer, network of computers, or other electronic devices, may resultin the one or more machines carrying out operations in accordance withembodiments described herein. A machine-readable medium may include, butis not limited to, floppy diskettes, optical disks, CD-ROMs (CompactDisc-Read Only Memories), and magneto-optical disks, ROMs, RAMs, EPROMs(Erasable Programmable Read Only Memories), EEPROMs (ElectricallyErasable Programmable Read Only Memories), magnetic or optical cards,flash memory, or other type of media/machine-readable medium suitablefor storing machine-executable instructions.

FIG. 6 illustrates an embodiment of a computing environment 600 capableof supporting the operations discussed above. The modules and systemscan be implemented in a variety of different hardware architectures andform factors including that shown in FIG. 5.

The Command Execution Module 601 includes a central processing unit tocache and execute commands and to distribute tasks among the othermodules and systems shown. It may include an instruction stack, a cachememory to store intermediate and final results, and mass memory to storeapplications and operating systems. The Command Execution Module mayalso serve as a central coordination and task allocation unit for thesystem.

The Screen Rendering Module 621 draws objects on the one or moremultiple screens for the user to see. It can be adapted to receive thedata from the Virtual Object Behavior Module 604, described below, andto render the virtual object and any other objects and forces on theappropriate screen or screens. Thus, the data from the Virtual ObjectBehavior Module would determine the position and dynamics of the virtualobject and associated gestures, forces and objects, for example, and theScreen Rendering Module would depict the virtual object and associatedobjects and environment on a screen, accordingly. The Screen RenderingModule could further be adapted to receive data from the Adjacent ScreenPerspective Module 607, described below, to either depict a targetlanding area for the virtual object if the virtual object could be movedto the display of the device with which the Adjacent Screen PerspectiveModule is associated. Thus, for example, if the virtual object is beingmoved from a main screen to an auxiliary screen, the Adjacent ScreenPerspective Module 2 could send data to the Screen Rendering Module tosuggest, for example in shadow form, one or more target landing areasfor the virtual object on that track to a user's hand movements or eyemovements.

The Object and Gesture Recognition Module 622 may be adapted torecognize and track hand and arm gestures of a user. Such a module maybe used to recognize hands, fingers, finger gestures, hand movements anda location of hands relative to displays. For example, the Object andGesture Recognition Module could for example determine that a user madea body part gesture to drop or throw a virtual object onto one or theother of the multiple screens, or that the user made a body part gestureto move the virtual object to a bezel of one or the other of themultiple screens. The Object and Gesture Recognition System may becoupled to a camera or camera array, a microphone or microphone array, atouch screen or touch surface, or a pointing device, or some combinationof these items, to detect gestures and commands from the user.

The touch screen or touch surface of the Object and Gesture RecognitionSystem may include a touch screen sensor. Data from the sensor may befed to hardware, software, firmware or a combination of the same to mapthe touch gesture of a user's hand on the screen or surface to acorresponding dynamic behavior of a virtual object. The sensor date maybe used to momentum and inertia factors to allow a variety of momentumbehavior for a virtual object based on input from the user's hand, suchas a swipe rate of a user's finger relative to the screen. Pinchinggestures may be interpreted as a command to lift a virtual object fromthe display screen, or to begin generating a virtual binding associatedwith the virtual object or to zoom in or out on a display. Similarcommands may be generated by the Object and Gesture Recognition Systemusing one or more cameras without the benefit of a touch surface.

The Direction of Attention Module 623 may be equipped with cameras orother sensors to track the position or orientation of a user's face orhands. When a gesture or voice command is issued, the system candetermine the appropriate screen for the gesture. In one example, acamera is mounted near each display to detect whether the user is facingthat display. If so, then the direction of attention module informationis provided to the Object and Gesture Recognition Module 622 to ensurethat the gestures or commands are associated with the appropriatelibrary for the active display. Similarly, if the user is looking awayfrom all of the screens, then commands can be ignored.

The Device Proximity Detection Module 625 can use proximity sensors,compasses, GPS (global positioning system) receivers, personal areanetwork radios, and other types of sensors, together with triangulationand other techniques to determine the proximity of other devices. Once anearby device is detected, it can be registered to the system and itstype can be determined as an input device or a display device or both.For an input device, received data may then be applied to the ObjectGesture and Recognition Module 622. For a display device, it may beconsidered by the Adjacent Screen Perspective Module 607.

The Virtual Object Behavior Module 604 is adapted to receive input fromthe Object Velocity and Direction Module, and to apply such input to avirtual object being shown in the display. Thus, for example, the Objectand Gesture Recognition System would interpret a user gesture and bymapping the captured movements of a user's hand to recognized movements,the Virtual Object Tracker Module would associate the virtual object'sposition and movements to the movements as recognized by Object andGesture Recognition System, the Object and Velocity and Direction Modulewould capture the dynamics of the virtual object's movements, and theVirtual Object Behavior Module would receive the input from the Objectand Velocity and Direction Module to generate data that would direct themovements of the virtual object to correspond to the input from theObject and Velocity and Direction Module.

The Virtual Object Tracker Module 606 on the other hand may be adaptedto track where a virtual object should be located in three-dimensionalspace in a vicinity of a display, and which body part of the user isholding the virtual object, based on input from the Object and GestureRecognition Module. The Virtual Object Tracker Module 606 may forexample track a virtual object as it moves across and between screensand track which body part of the user is holding that virtual object.Tracking the body part that is holding the virtual object allows acontinuous awareness of the body part's air movements, and thus aneventual awareness as to whether the virtual object has been releasedonto one or more screens.

The Gesture to View and Screen Synchronization Module 608, receives theselection of the view and screen or both from the Direction of AttentionModule 623 and, in some cases, voice commands to determine which view isthe active view and which screen is the active screen. It then causesthe relevant gesture library to be loaded for the Object and GestureRecognition Module 622. Various views of an application on one or morescreens can be associated with alternative gesture libraries or a set ofgesture templates for a given view. As an example, in FIG. 1A, apinch-release gesture launches a torpedo, but in FIG. 1B, the samegesture launches a depth charge.

The Adjacent Screen Perspective Module 607, which may include or becoupled to the Device Proximity Detection Module 625, may be adapted todetermine an angle and position of one display relative to anotherdisplay. A projected display includes, for example, an image projectedonto a wall or screen. The ability to detect a proximity of a nearbyscreen and a corresponding angle or orientation of a display projectedtherefrom may for example be accomplished with either an infraredemitter and receiver, or electromagnetic or photo-detection sensingcapability. For technologies that allow projected displays with touchinput, the incoming video can be analyzed to determine the position of aprojected display and to correct for the distortion caused by displayingat an angle. An accelerometer, magnetometer, compass, or camera can beused to determine the angle at which a device is being held whileinfrared emitters and cameras could allow the orientation of the screendevice to be determined in relation to the sensors on an adjacentdevice. The Adjacent Screen Perspective Module 607 may, in this way,determine coordinates of an adjacent screen relative to its own screencoordinates. Thus, the Adjacent Screen Perspective Module may determinewhich devices are in proximity to each other, and further potentialtargets for moving one or more virtual objects across screens. TheAdjacent Screen Perspective Module may further allow the position of thescreens to be correlated to a model of three-dimensional spacerepresenting all of the existing objects and virtual objects.

The Object and Velocity and Direction Module 603 may be adapted toestimate the dynamics of a virtual object being moved, such as itstrajectory, velocity (whether linear or angular), momentum (whetherlinear or angular), etc. by receiving input from the Virtual ObjectTracker Module. The Object and Velocity and Direction Module may furtherbe adapted to estimate dynamics of any physics forces, by for exampleestimating the acceleration, deflection, degree of stretching of avirtual binding, etc. and the dynamic behavior of a virtual object oncereleased by a user's body part. The Object and Velocity and DirectionModule may also use image motion, size and angle changes to estimate thevelocity of objects, such as the velocity of hands and fingers

The Momentum and Inertia Module 602 can use image motion, image size,and angle changes of objects in the image plane or in athree-dimensional space to estimate the velocity and direction ofobjects in the space or on a display. The Momentum and Inertia Module iscoupled to the Object and Gesture Recognition Module 622 to estimate thevelocity of gestures performed by hands, fingers, and other body partsand then to apply those estimates to determine momentum and velocitiesto virtual objects that are to be affected by the gesture.

The 3D Image Interaction and Effects Module 605 tracks user interactionwith 3D images that appear to extend out of one or more screens. Theinfluence of objects in the z-axis (towards and away from the plane ofthe screen) can be calculated together with the relative influence ofthese objects upon each other. For example, an object thrown by a usergesture can be influenced by 3D objects in the foreground before thevirtual object arrives at the plane of the screen. These objects maychange the direction or velocity of the projectile or destroy itentirely. The object can be rendered by the 3D Image Interaction andEffects Module in the foreground on one or more of the displays. Asillustrated, various components, such as components 601, 602, 603, 604,605, 606, 607, and 608 are connected via an interconnect or a bus, suchas bus 609.

The following clauses and/or examples pertain to further embodiments orexamples. Specifics in the examples may be used anywhere in one or moreembodiments. The various features of the different embodiments orexamples may be variously combined with some features included andothers excluded to suit a variety of different applications. Examplesmay include subject matter such as a method, means for performing actsof the method, at least one machine-readable medium includinginstructions that, when performed by a machine cause the machine toperform acts of the method, or of an apparatus or system forfacilitating hybrid communication according to embodiments and examplesdescribed herein.

Some embodiments pertain to Example 1 that includes an apparatus tofacilitate deep learning-based real-time detection and correction ofcompromised sensors in autonomous machines, the apparatus comprising:detection and capturing logic to facilitate one or more sensors tocapture one or more images of a scene, wherein an image of the one ormore images is determined to be unclear, wherein the one or more sensorsinclude one or more cameras; and classification and prediction logic tofacilitate a deep learning model to identify, in real-time, a sensorassociated with the image.

Example 2 includes the subject matter of Example 1, further comprisingconcatenation logic to receive one or more data inputs associated withthe one or more images to concatenate the one or more data inputs into asingle data input to be processed by the deep learning model, whereinthe apparatus comprises an autonomous machine includes one or more of aself-driving vehicle, a self-flying vehicle, a self-sailing vehicle, andan autonomous household device.

Example 3 includes the subject matter of Examples 1-2, furthercomprising training and inference logic to facilitate the deep learningmodel to receive the single data input to perform one or more deeplearning processes including a training process and an inferencingprocess to obtain real-time identification of the sensor associated withthe unclear image, wherein the sensor includes a camera.

Example 4 includes the subject matter of Examples 1-3, wherein thetraining and inferencing logic is further to facilitate the deeplearning model to receive a plurality of data inputs and run theplurality of data inputs through the training and inferencing processessuch that the real-time identification of the sensor is accurate andtimely.

Example 5 includes the subject matter of Examples 1-4, wherein the deeplearning model comprises one or more neural networks including one ormore convolutional neural networks, wherein the image is unclear due toone or more of a technical defect with the sensor or a physicalobstruction of the sensors, wherein the physical obstruction is due to aperson, a plant, an animal, or an object obstructing the sensor, ordirt, stains, mud, or debris covering a portion of a lens of the sensor.

Example 6 includes the subject matter of Examples 1-5, wherein theclarification and prediction logic to provide one or more of real-timenotification of the unclear image, and real-time auto-correction of thesensor.

Example 7 includes the subject matter of Examples 1-6, wherein theapparatus comprises one or more processors having a graphics processorco-located with an application processor on a common semiconductorpackage.

Some embodiments pertain to Example 8 that includes a methodfacilitating deep learning-based real-time detection and correction ofcompromised sensors in autonomous machines, the method comprising:facilitating one or more sensors to capture one or more images of ascene, wherein an image of the one or more images is determined to beunclear, wherein the one or more sensors include one or more cameras ofa computing device; and facilitating a deep learning model to identify,in real-time, a sensor associated with the image.

Example 9 includes the subject matter of Example 8, further comprisingreceiving one or more data inputs associated with the one or more imagesto concatenate the one or more data inputs into a single data input tobe processed by the deep learning model, wherein the apparatus comprisesan autonomous machine includes one or more of a self-driving vehicle, aself-flying vehicle, a self-sailing vehicle, and an autonomous householddevice.

Example 10 includes the subject matter of Examples 8-9, furthercomprising facilitating the deep learning model to receive the singledata input to perform one or more deep learning processes including atraining process and an inferencing process to obtain real-timeidentification of the sensor associated with the unclear image, whereinthe sensor includes a camera.

Example 11 includes the subject matter of Examples 8-10, wherein thedeep learning model is further to receive a plurality of data inputs andrun the plurality of data inputs through the training and inferencingprocesses such that the real-time identification of the sensor isaccurate and timely.

Example 12 includes the subject matter of Examples 8-11, wherein thedeep learning model comprises one or more neural networks including oneor more convolutional neural networks, wherein the image is unclear dueto one or more of a technical defect with the sensor or a physicalobstruction of the sensors, wherein the physical obstruction is due to aperson, a plant, an animal, or an object obstructing the sensor, ordirt, stains, mud, or debris covering a portion of a lens of the sensor.

Example 13 includes the subject matter of Examples 8-12, furthercomprising providing one or more of real-time notification of theunclear image, and real-time auto-correction of the sensor.

Example 14 includes the subject matter of Examples 8-13, wherein thecomputing device comprises one or more processors having a graphicsprocessor co-located with an application processor on a commonsemiconductor package.

Some embodiments pertain to Example 15 that includes a data processingsystem comprising a computing device having memory coupled to aprocessing device, the processing device to: facilitate one or moresensors to capture one or more images of a scene, wherein an image ofthe one or more images is determined to be unclear, wherein the one ormore sensors include one or more cameras of a computing device; andfacilitate a deep learning model to identify, in real-time, a sensorassociated with the image.

Example 16 includes the subject matter of Example 15, wherein theprocessing device is further to receive one or more data inputsassociated with the one or more images to concatenate the one or moredata inputs into a single data input to be processed by the deeplearning model, wherein the apparatus comprises an autonomous machineincludes one or more of a self-driving vehicle, a self-flying vehicle, aself-sailing vehicle, and an autonomous household device.

Example 17 includes the subject matter of Examples 15-16, wherein theprocessing device is further to facilitate the deep learning model toreceive the single data input to perform one or more deep learningprocesses including a training process and an inferencing process toobtain real-time identification of the sensor associated with theunclear image, wherein the sensor includes a camera.

Example 18 includes the subject matter of Examples 15-17, wherein thedeep learning model is further to receive a plurality of data inputs andrun the plurality of data inputs through the training and inferencingprocesses such that the real-time identification of the sensor isaccurate and timely.

Example 19 includes the subject matter of Examples 15-18, wherein thedeep learning model comprises one or more neural networks including oneor more convolutional neural networks, wherein the image is unclear dueto one or more of a technical defect with the sensor or a physicalobstruction of the sensors, wherein the physical obstruction is due to aperson, a plant, an animal, or an object obstructing the sensor, ordirt, stains, mud, or debris covering a portion of a lens of the sensor.

Example 20 includes the subject matter of Examples 15-19, wherein theprocessing device is further to provide one or more of real-timenotification of the unclear image, and real-time auto-correction of thesensor.

Example 21 includes the subject matter of Examples 15-20, wherein thecomputing device comprises one or more processors having a graphicsprocessor co-located with an application processor on a commonsemiconductor package.

Some embodiments pertain to Example 22 that includes an apparatus tofacilitate simultaneous recognition and processing of multiple speechesfrom multiple users, the apparatus comprising: means for facilitatingone or more sensors to capture one or more images of a scene, wherein animage of the one or more images is determined to be unclear, wherein theone or more sensors include one or more cameras; and means forfacilitating a deep learning model to identify, in real-time, a sensorassociated with the image.

Example 23 includes the subject matter of Example 22, further comprisingmeans for receiving one or more data inputs associated with the one ormore images to concatenate the one or more data inputs into a singledata input to be processed by the deep learning model, wherein theapparatus comprises an autonomous machine includes one or more of aself-driving vehicle, a self-flying vehicle, a self-sailing vehicle, andan autonomous household device.

Example 24 includes the subject matter of Examples 22-23, furthercomprising means for facilitating the deep learning model to receive thesingle data input to perform one or more deep learning processesincluding a training process and an inferencing process to obtainreal-time identification of the sensor associated with the unclearimage, wherein the sensor includes a camera.

Example 25 includes the subject matter of Examples 22-24, wherein thedeep learning model is further to receive a plurality of data inputs andrun the plurality of data inputs through the training and inferencingprocesses such that the real-time identification of the sensor isaccurate and timely.

Example 26 includes the subject matter of Examples 22-25, wherein thedeep learning model comprises one or more neural networks including oneor more convolutional neural networks, wherein the image is unclear dueto one or more of a technical defect with the sensor or a physicalobstruction of the sensors, wherein the physical obstruction is due to aperson, a plant, an animal, or an object obstructing the sensor, ordirt, stains, mud, or debris covering a portion of a lens of the sensor.

Example 27 includes the subject matter of Examples 22-26, furthercomprising means for providing one or more of real-time notification ofthe unclear image, and real-time auto-correction of the sensor.

Example 28 includes the subject matter of Examples 22-27, wherein theapparatus comprises one or more processors having a graphics processorco-located with an application processor on a common semiconductorpackage.

Example 29 includes at least one non-transitory or tangiblemachine-readable medium comprising a plurality of instructions, whenexecuted on a computing device, to implement or perform a method asclaimed in any of claims or examples 8-14.

Example 30 includes at least one machine-readable medium comprising aplurality of instructions, when executed on a computing device, toimplement or perform a method as claimed in any of claims or examples8-14.

Example 31 includes a system comprising a mechanism to implement orperform a method as claimed in any of claims or examples 8-14.

Example 32 includes an apparatus comprising means for performing amethod as claimed in any of claims or examples 8-14.

Example 33 includes a computing device arranged to implement or performa method as claimed in any of claims or examples 8-14.

Example 34 includes a communications device arranged to implement orperform a method as claimed in any of claims or examples 8-14.

Example 35 includes at least one machine-readable medium comprising aplurality of instructions, when executed on a computing device, toimplement or perform a method or realize an apparatus as claimed in anypreceding claims.

Example 36 includes at least one non-transitory or tangiblemachine-readable medium comprising a plurality of instructions, whenexecuted on a computing device, to implement or perform a method orrealize an apparatus as claimed in any preceding claims.

Example 37 includes a system comprising a mechanism to implement orperform a method or realize an apparatus as claimed in any precedingclaims.

Example 38 includes an apparatus comprising means to perform a method asclaimed in any preceding claims.

Example 39 includes a computing device arranged to implement or performa method or realize an apparatus as claimed in any preceding claims.

Example 40 includes a communications device arranged to implement orperform a method or realize an apparatus as claimed in any precedingclaims.

The drawings and the forgoing description give examples of embodiments.Those skilled in the art will appreciate that one or more of thedescribed elements may well be combined into a single functionalelement. Alternatively, certain elements may be split into multiplefunctional elements. Elements from one embodiment may be added toanother embodiment. For example, orders of processes described hereinmay be changed and are not limited to the manner described herein.Moreover, the actions of any flow diagram need not be implemented in theorder shown; nor do all of the acts necessarily need to be performed.Also, those acts that are not dependent on other acts may be performedin parallel with the other acts. The scope of embodiments is by no meanslimited by these specific examples. Numerous variations, whetherexplicitly given in the specification or not, such as differences instructure, dimension, and use of material, are possible. The scope ofembodiments is at least as broad as given by the following claims.

What is claimed is:
 1. An apparatus comprising: detection and capturinglogic to facilitate one or more sensors to capture one or more images ofa scene, wherein an image of the one or more images is determined to beunclear, wherein the one or more sensors include one or more cameras;and classification and prediction logic to facilitate a deep learningmodel to identify, in real-time, a sensor associated with the image. 2.The apparatus of claim 1, further comprising concatenation logic toreceive one or more data inputs associated with the one or more imagesto concatenate the one or more data inputs into a single data input tobe processed by the deep learning model, wherein the apparatus comprisesan autonomous machine includes one or more of a self-driving vehicle, aself-flying vehicle, a self-sailing vehicle, and an autonomous householddevice.
 3. The apparatus of claim 1, further comprising training andinference logic to facilitate the deep learning model to receive thesingle data input to perform one or more deep learning processesincluding a training process and an inferencing process to obtainreal-time identification of the sensor associated with the unclearimage, wherein the sensor includes a camera.
 4. The apparatus of claim3, wherein the training and inferencing logic is further to facilitatethe deep learning model to receive a plurality of data inputs and runthe plurality of data inputs through the training and inferencingprocesses such that the real-time identification of the sensor isaccurate and timely.
 5. The apparatus of claim 1, wherein the deeplearning model comprises one or more neural networks including one ormore convolutional neural networks, wherein the image is unclear due toone or more of a technical defect with the sensor or a physicalobstruction of the sensors, wherein the physical obstruction is due to aperson, a plant, an animal, or an object obstructing the sensor, ordirt, stains, mud, or debris covering a portion of a lens of the sensor.6. The apparatus of claim 1, wherein the clarification and predictionlogic to provide one or more of real-time notification of the unclearimage, and real-time auto-correction of the sensor.
 7. The apparatus ofclaim 1, wherein the apparatus comprises one or more processors having agraphics processor co-located with an application processor on a commonsemiconductor package.
 8. A method comprising: facilitating one or moresensors to capture one or more images of a scene, wherein an image ofthe one or more images is determined to be unclear, wherein the one ormore sensors include one or more cameras of a computing device; andfacilitating a deep learning model to identify, in real-time, a sensorassociated with the image.
 9. The method of claim 8, further comprisingreceiving one or more data inputs associated with the one or more imagesto concatenate the one or more data inputs into a single data input tobe processed by the deep learning model, wherein the apparatus comprisesan autonomous machine includes one or more of a self-driving vehicle, aself-flying vehicle, a self-sailing vehicle, and an autonomous householddevice.
 10. The method of claim 8, further comprising facilitating thedeep learning model to receive the single data input to perform one ormore deep learning processes including a training process and aninferencing process to obtain real-time identification of the sensorassociated with the unclear image, wherein the sensor includes a camera.11. The method of claim 10, wherein the deep learning model is furtherto receive a plurality of data inputs and run the plurality of datainputs through the training and inferencing processes such that thereal-time identification of the sensor is accurate and timely.
 12. Themethod of claim 8, wherein the deep learning model comprises one or moreneural networks including one or more convolutional neural networks,wherein the image is unclear due to one or more of a technical defectwith the sensor or a physical obstruction of the sensors, wherein thephysical obstruction is due to a person, a plant, an animal, or anobject obstructing the sensor, or dirt, stains, mud, or debris coveringa portion of a lens of the sensor.
 13. The method of claim 8, furthercomprising providing one or more of real-time notification of theunclear image, and real-time auto-correction of the sensor.
 14. Themethod of claim 8, wherein the computing device comprises one or moreprocessors having a graphics processor co-located with an applicationprocessor on a common semiconductor package.
 15. At least onemachine-readable medium comprising instructions which, when executed bya computing device, cause the computing device to perform operationscomprising: facilitating one or more sensors to capture one or moreimages of a scene, wherein an image of the one or more images isdetermined to be unclear, wherein the one or more sensors include one ormore cameras; and facilitating a deep learning model to identify, inreal-time, a sensor associated with the image.
 16. The machine-readablemedium of claim 15, wherein the operations further comprise receivingone or more data inputs associated with the one or more images toconcatenate the one or more data inputs into a single data input to beprocessed by the deep learning model, wherein the apparatus comprises anautonomous machine includes one or more of a self-driving vehicle, aself-flying vehicle, a self-sailing vehicle, and an autonomous householddevice.
 17. The machine-readable medium of claim 15, wherein theoperations further comprise facilitating the deep learning model toreceive the single data input to perform one or more deep learningprocesses including a training process and an inferencing process toobtain real-time identification of the sensor associated with theunclear image, wherein the sensor includes a camera.
 18. Themachine-readable medium of claim 17, wherein the deep learning model isfurther to receive a plurality of data inputs and run the plurality ofdata inputs through the training and inferencing processes such that thereal-time identification of the sensor is accurate and timely.
 19. Themachine-readable medium of claim 15, wherein the deep learning modelcomprises one or more neural networks including one or moreconvolutional neural networks, wherein the image is unclear due to oneor more of a technical defect with the sensor or a physical obstructionof the sensors, wherein the physical obstruction is due to a person, aplant, an animal, or an object obstructing the sensor, or dirt, stains,mud, or debris covering a portion of a lens of the sensor.
 20. Themachine-readable medium of claim 15, wherein the operations furthercomprise providing one or more of real-time notification of the unclearimage, and real-time auto-correction of the sensor, wherein thecomputing device comprises one or more processors having a graphicsprocessor co-located with an application processor on a commonsemiconductor package.